Skip to content

[ops] fix: keep verl FusedLinearForPPO batch-invariant under Liger - #48

Closed
TimYangst wants to merge 1 commit into
verl-project:mainfrom
TimYangst:tingyang/fix/batch-invariant-verl-liger
Closed

TimYangst wants to merge 1 commit into
verl-project:mainfrom
TimYangst:tingyang/fix/batch-invariant-verl-liger

Conversation

@TimYangst

Copy link
Copy Markdown
Contributor

verl-project/verl#7461 makes FusedLinearForPPO route the fused output head through Liger's LigerFusedLinearScaledCrossEntropyFunction whenever liger_kernel is importable. That kernel bypasses vexact's aten::mm override, so training-side log-probs stop matching inference log-probs bit-for-bit: test_fused_linear_for_ppo_log_probs_float32_in_bfloat16_model fails with 98.9% mismatched elements (max abs diff 0.6875) on verl 8e4a572 + liger 0.8.2.

While batch-invariant mode is on, null verl's module-level dispatch handle so FusedLinearForPPO falls back to its chunked torch.matmul + cross-entropy path (which the overrides cover), and restore it when the mode is turned off. The verl integration imports verl's module before enabling the mode so the hook always lands, and enable_batch_invariant_mode() re-applies it when the mode was already on. Add a test for the toggle.

Verified on 1xH100 with verl 8e4a572 + liger-kernel 0.8.2: tests/batch_invariant_ops/ 68 passed (was 2 failed).

What does this PR do?

Concise overview of the change. Reference related issues/PRs.

Checklist Before Starting

  • Search for relative PRs/issues and link here: ...
  • PR title follows [{modules}] {type}: {description} format (see check_pr_title.py for the full list of allowed modules and types)
    • Breaking changes: prepend [BREAKING] — e.g. [BREAKING][ops] feat: new batch-invariant matmul API

Test

Validation results (numeric checks, benchmark metrics) for changes not covered by CI.

API and Usage Example

Show API changes and usage examples if applicable.

Design & Code Changes

High-level design description and specific change list.

Checklist Before Submitting

  • Read the Contribute Guide
  • Applied pre-commit checks (pre-commit run --all-files)
  • Added/updated documentation
  • Added tests to CI workflow (or explained why not feasible)

verl-project/verl#7461 makes FusedLinearForPPO route the fused output head
through Liger's LigerFusedLinearScaledCrossEntropyFunction whenever
liger_kernel is importable. That kernel bypasses vexact's aten::mm override,
so training-side log-probs stop matching inference log-probs bit-for-bit:
test_fused_linear_for_ppo_log_probs_float32_in_bfloat16_model fails with
98.9% mismatched elements (max abs diff 0.6875) on verl 8e4a572 + liger 0.8.2.

While batch-invariant mode is on, null verl's module-level dispatch handle so
FusedLinearForPPO falls back to its chunked torch.matmul + cross-entropy path
(which the overrides cover), and restore it when the mode is turned off. The
verl integration imports verl's module before enabling the mode so the hook
always lands, and enable_batch_invariant_mode() re-applies it when the mode
was already on. Add a test for the toggle.

Verified on 1xH100 with verl 8e4a572 + liger-kernel 0.8.2:
tests/batch_invariant_ops/ 68 passed (was 2 failed).
@Luosuu

Luosuu commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

please make PR to verl to avoid monkeypatch

@Luosuu Luosuu closed this Sep 2, 2026
wuxibin89 pushed a commit to verl-project/verl that referenced this pull request Sep 3, 2026
### What does this PR do?

Makes `fused_kernel_options.impl_backend` authoritative for the PPO
output head:

- `torch` always uses verl's native chunked `FusedLinearForPPOFunction`.
- `liger` explicitly opts into
`LigerFusedLinearScaledCrossEntropyFunction` and retains the native
fallback when Liger is unavailable.
- The selected backend is propagated through all five Hugging Face
fused-forward adapters.

This fixes a regression introduced by #7461. That change selected Liger
whenever it was importable, even when the configured backend was
`torch`. Besides violating the backend contract, the implicit Liger
dispatch bypasses PyTorch operator overrides used by exactness
integrations such as vexact, forcing vexact to modify verl's private
module state as proposed in verl-project/vexact#48.

This does not duplicate an existing PR. Before implementation, open PRs
and issues were searched for `FusedLinearForPPO Liger batch invariant`,
`LigerFusedLinearScaledCrossEntropyFunction`, `fused linear PPO
exactness`, and `disable Liger fused linear`; no matching open work was
found.

AI assistance: OpenAI Codex assisted with implementation and test
execution. The human submitter reviewed every changed line and
understands the change end-to-end. The relevant tests listed below were
run in the shared development workspace before submission.

### Test

- `python -m pytest -q
tests/utils/test_experimental_torch_functional_on_cpu.py` — 6 passed.
- `python -m pytest -q
tests/models/test_fused_kernels_ulysses_sp_on_cpu.py` — 26 passed.
- `pre-commit run --files <all changed files>` — all hooks passed.
- vexact `test_fused_linear_for_ppo_batch_invariance_forward` on 1x H20,
without a vexact monkey patch or environment override — 2 passed.

Additional H20 diagnostic: the full vexact file has two existing strict
bitwise comparison failures between the native chunked 2D matmul path
and the 3D reference matmul, with maximum absolute differences of
`1.52587890625e-05` and `1.9073486328125e-06`. This is separate from the
implicit Liger dispatch fixed here; verl-project/vexact#48 reports 68
passing tests on H100 when using the same native fallback path.

### API and Usage Example

The default remains the native torch backend:

```yaml
actor_rollout_ref:
  model:
    use_fused_kernels: true
    fused_kernel_options:
      impl_backend: torch
```

Select Liger explicitly when desired:

```yaml
actor_rollout_ref:
  model:
    use_fused_kernels: true
    fused_kernel_options:
      impl_backend: liger
```

### Design & Code Changes

- Add an explicit `impl_backend` argument to `FusedLinearForPPO`,
defaulting to `torch`.
- Route to Liger only when `impl_backend="liger"`.
- Store the selected fused backend on each model instance and pass it
through dense, Qwen2-VL, Qwen3-VL, Qwen3.5, and GLM4V torch forward
adapters.
- Document `liger` as an explicit backend and clarify the `torch`
backend behavior.
- Add unit coverage for backend dispatch, validation, and adapter
propagation.

### Checklist Before Submitting

- [x] Read the contribution and agent instructions.
- [x] Search for duplicate issues and PRs.
- [x] Add/update tests and documentation.
- [x] Run pre-commit checks.
- [x] Disclose AI assistance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants